As a fundamental issue in lifelong learning, catastrophic forgetting is directly caused by inaccessible historical data; accordingly, if the data (information) were memorized perfectly, no forgetting should be expected. Motivated by that, we propose a GAN memory for lifelong learning, which is capable of remembering a stream of datasets via generative processes, with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the ``style'' of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. The GAN memory -- that is motivated by lifelong learning -- is therefore itself manifested by a form of lifelong learning, via forward transfer and modulation of information from prior tasks. Experiments demonstrate the superiority of our method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems.

Figure 7: (a) The generator architecture adopted in this paper. It inherits the architecture from GP-GAN which is the same as the green/frozen part of our GAN memory (see Figure 1(a)). Given a target task/data ( e.g., Flowers, Cathedrals, or Cats), all the parameters are trainable and fine-tuned to fit the target data. FC/Conv layer, one need to apply it to all layers in real implementation. "None" means no modulation from target data is applied, we only use the modulation from the Only" means that we replace the Only" means that we replace the Only" means that we replace the b "All" means all style parameters from target data "All" is obtained via a similar way to that of Figure 2(a); " means using the style parameters from target data without "None" and "All" are obtained via a similar way to that of Figure 2(a); "FC" is obtained by applying a newly designed style parameters which copies the style parameters " is obtained by copying the designed style parameters under the "FC" setting and replacing " is obtained by copying the designed style parameters under the " "Our" is our GAN memory; "NoNorm" is a modified version of our GAN memory which removes the normalization on "NoBias" is a modified version of our GAN memory which removes the bias term Here we discuss the detailed techniques for interpolation among different generative processes with our GAN memory and show more examples in Figure 8, 9, 10, and 11.

That revelation is anticipated, for if the characteristics of previous data are remembered perfectly ( e.g., via realistic generative replay), no forgetting should be expected for lifelong learning.

By "well-behaved" we mean the shape within kernels (see L175; shared between source

Figure 7: (a) The generator architecture adopted in this paper. It inherits the architecture from GP-GAN which is the same as the green/frozen part of our GAN memory (see Figure 1(a)). Given a target task/data ( e.g., Flowers, Cathedrals, or Cats), all the parameters are trainable and fine-tuned to fit the target data. FC/Conv layer, one need to apply it to all layers in real implementation. "None" means no modulation from target data is applied, we only use the modulation from the Only" means that we replace the Only" means that we replace the Only" means that we replace the b "All" means all style parameters from target data "All" is obtained via a similar way to that of Figure 2(a); " means using the style parameters from target data without "None" and "All" are obtained via a similar way to that of Figure 2(a); "FC" is obtained by applying a newly designed style parameters which copies the style parameters " is obtained by copying the designed style parameters under the "FC" setting and replacing " is obtained by copying the designed style parameters under the " "Our" is our GAN memory; "NoNorm" is a modified version of our GAN memory which removes the normalization on "NoBias" is a modified version of our GAN memory which removes the bias term Here we discuss the detailed techniques for interpolation among different generative processes with our GAN memory and show more examples in Figure 8, 9, 10, and 11.

Additional Feedback: * Important technical details do need to be in the main paper. For example, the approach and the protocol for incremental learning with replay should be in the main paper, at least briefly. The current level of detail for this is insufficient (some of this is in the supplementary material, but it is really needed in the main paper). For example, if the sequential tasks were based on data from cat, dog, horse, and cow classes, how would the generated samples be? Is the model robust to such data?

As a fundamental issue in lifelong learning, catastrophic forgetting is directly caused by inaccessible historical data; accordingly, if the data (information) were memorized perfectly, no forgetting should be expected. Motivated by that, we propose a GAN memory for lifelong learning, which is capable of remembering a stream of datasets via generative processes, with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the style'' of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. The GAN memory -- that is motivated by lifelong learning -- is therefore itself manifested by a form of lifelong learning, via forward transfer and modulation of information from prior tasks.